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ABSTRACT 

This research examines the subject of practice and 
transfer effects using an intelligent tutoring system (ITS) teaching 
novel knowledge and skills (i.e., flight engineering). Previous 
research has shown* that when more time is spent exercising new 
cognitive skills, performance is improved and the cognitive load is 
reduced. Other research has demonstrated that when the number or 
variety of example problems is small, learning tends to be rapid, but 
transfer tends to be weak. The purpose of this research was to test 
these ideas in a controlled setting using an ITS that was manipulated 
to yield 2 contrasting learning environments: "extended" (12 problems 
per problem set), and "constrained" (3 problems per problem set). 
Subjects were 356 male and female adult high school graduates without 
any experience in flight engineering. The environments differed only 
in the number of practice problems requiring solution in the various 
problem sets. Results show that while subjects in the constrained 
environment completed the curriculum significantly faster than did 
subjects in the extended version, there were no differences between 
conditions on any of the outcome measures. But when the data were 
examined across problem sets, latency and error-type differences 
between the two groups were found. Three figures and one table 
present study findings. (Contains 8 references.) (Author/SLD) 
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IF PRACTICE MAKES PERFECT, 
WHAT DOES LESS PRACTICE MAKE? 



SUMMARY 

"Practice makes perfect"--more time spent exercising new cognitive skills 
results in improved performance and reduced cognitive load (e.g., Ackerman 
1988; Anderson. 1987; Schneider & Shiffrin. 1977). In addition, when the 
number (or variety) of example problems is small, learning tends to be rapid, 
but transfer tends to be weak (e.g.. Carlson & Yaure. 1990; Gick & Holyoak, 
1987). This paper examined these issues in a controlled setting using an 
intelligent tutoring system teaching novel knowledge and skills (i.e.. flight 
engineering). We manipulated the tutor to yield two contrasting learning 
environments: "extended" (12 problems per problem set) and "constrained" (3 
problems per problem set). These environments differed only in the number 
of practice problems requiring solution In the various problem sets. While 
subjects in the shorter version were expected to complete the curriculum faster, 
we wanted to examine practice effects on learning outcome (transfer) and skill 
acquisition (error and latency analyses). Results showed that while subjects 
in the constrained environment completed the curriculum significantly faster than 
subjects in the extended version, there were no differences on any of the 
outcome tests. Moreover, when the data were examined across problem sets, 
latency and error-type differences between the two groups were found. 



INTRODUCTION 

An enduring expression that has a lot of empirical support in cognitive 
psychology is that "practice makes perfect" (e.g., Ackerman, 1988; Anderson, 
1987; Fisk, in press; Schneider & Shiffrin, 1977; Woltz, 1988). Increasing the 
time allocated to exercising a new co9nitive skill typically results in improved 
performance and a reduced cognitive load. A related finding is that when the 
number (or variety) of example problems is small, learning tends to be rapid, 
but transfer tends to be weak (e.g., Carlson & Yaure, 1990; Gick & Holyoak, 
1987). Thus, more and varied practice problems lead to more effective skill 
acquisition. But how many practice problems are sufficient? Alternatively, how 
few practice problems can be provided that will yield comparable learning 
outcomes to richer practice environments? 

The purpose of this paper is to investigate the effects of practice on learning 
outcome. Other questions address the relationship between practice effects and 
the learning process. In particular, what are the effects of practice on learning 
curves and errors made during skill acquisition? For example, do subjects 
tend to compensate for fewer practice problems by adjusting (increasing) their 
time spent learning each problem? Is there an effect of practice on the number 
or nature of errors made during the learning process? 



ERIC 



7 

1 



We will present results from a large-scale experiment that was designed to 
test practice effects from a systematically altered tutoring system teaching flight 
engineering skills. We manipulated the tutor to yield two rontrasting learning 
environments, differing only in the number of problems the learner needed to 
solve in each of nine different problem sets. The version with many problems 
was called "extended" (12 problems per problem set). The version with fewer 
problems to solve was called "constrained" (3 problems per set). A fixed ratio 
of 4:1 existed between the two tutor versions. 

The simple hypotheses were that (a) subjects assigned to the constrained 
environment should take less time to complete the tutor because there were 
considerably fewer problems for them to solve. However, (b) these same 
subjects were not expected to perform as well on the posttests compared with 
subjects learning from the extended environment who would have received 
considerably more practice solving tutor-related problems. In addition, we 
expected that (c) subjects would tend to adjust their learning times per problem 
where those in the short version would, over time, take longer to solve each 
of their three problems (per problem set) compared to subjects In the longer 
version. This adjustment was expected to occur gradually, as learners realized 
they had fewer problems and thus invested more time per problem. Finally, 
we hypothesized that (d) subjects in the constrained version would manifest 
more conceptual (rather than computational) errors during learning given their 
sparse practice environment. 



METHOD 



Subjects 

The subjects in this study consisted of 356 males and females participating 
in a seven-day study on the acquisition of flight engineering knowledge and 
skills from an intelligent tutoring system. The gender distribution in the sample 
was approximately 75 percent males and 25 percent vemales. All subjects 
were high school graduates with a mean age of 22 years. Subjects were 
obtained from a local temporary employment agency and were paid for their 
participation, consisting of forty-five hours of testing and learning. None of the 
subjects had any prior experience or training as flight engineers or pilots. 



Flight Engineering Tutor 

The tutor was originally developed at the University of Pittsburgh (Lesgold, 
Bunzo, & Eastman, 1989) and then modified at the Armstrong l-aboratory to fit 
experimental objectives. The tutor was designed to teach knowledge and skills 
associated with a flight engineer's job. Job components include collecting and 
analyzing Information about a pending flight and deciding whether various factors 
(e.g., weather and runway conditions, type of aircraft) indicate a safe flight. 
There were two parts to the cun-iculum: graph reading, and computing values 
for inclusion in the TOLD (Take Off and Landing Data) sheet. The focus of 




?]nr^3£!L« 0" performance within the second, more substantive, part of the 
tutor dealing with completing the TOLD worksheet. 

anrl^JnwI^f* Worksheet part of the curriculum consisted of nine problem sets 
!,nlc^oJt applying the graph reading skills taught earlier to fiil in the TOLD 
ht LhT^® wo^'^^heet was designed to match the actual forms 

nrnnpr L^?^* engineers. In formation must be assembled and entered into the 
proper cells. Some information was given to the learners (e.g.. gross weiqht 
of the aircraft, wind direction and velocity, the length, heading; and wnSs 

derived froT^rnm^ii^''r L" P^*^)- O^h^^ haTfo bl 

u,nrucl^ ir°°"^P'? entered into the correct cells of the 

h2JS!^f' ♦ .^9^ '"Stance, the wind-components chart is used to compute the 
?h«2^^ " JSi'^'^'i ? crosswind components, and consists of two superimposed 
£«m?S^n^"^,P^'^®*t'^" coordinates. On-line tools were available to ass^t 
iJ u in making the necessary computations. Some of the tools that 
could be used with the wind-components chart included draw vertical o 

?or5f°"*A a^f'-^^^w^ ^^1"^ °^ erase lines, redi^splay grapl^, and so 

forth A Help" window allowed the learner to read related information aboiS 
the topic under study. Learning was self-paced. '"lormanon aoout 

thP vS;??J®1« ®® K?^*^ of problems (sets 1, 2, and 3) involved computing 
the maximum allowable crosswind given the gross weiqht of the aircraft and 

mnr:"3r\'°"5''°" ^^'^ problem serbeime pmg^^^^^^^ 

or L^S^lc to the problem-solving phase in each problem set, conce^s 
ti^o? HoS. *° discussed and illustrated by the 

nfr? th2 •• f^°^exa["P"e. the tutor explicitly demonstrated how to figure 
plre 1^ thf "'f ^ f T'^^^ P^^senting any problems (sel 

?S Ji '^"^i °hart was displayed on the Computer screen 

tlTp ni!«n ^'"P"*®' •^i;!'' / "P fre"^ the X-axis (corresponding to 

the given gross weight of the aircraft) to intersect the appropriate RCR A 
fhi^w ^fP'^ye^ moving horizontally across from the RCR curve to 

the y-axis. The intersection at the y-axis yielded the crosswind value TeS 

l^ttrf 7 ^^OSSWIND FOR TAKEOFF chart is shown below. To use the 
^^f'^Ji"^^ ^^''^l^s are needed: (1) Gross weight of L aircraft 

^'^^^^^ ''^^ '^^ measur:ofthe%r^.lf:;^^ 

hooI!!IinH^®°°"^ °I P^°'''®'" sets (4, 5, and 6) involved computing the 

KadnrS;°^T"^'. taiiwind components, respectively. HerersubjeSI 
had to apply rules learned earlier about what constituted various wind tvo^ 

dfo^pp^^Vio^^f^SS^ '"'^ " '^^^^^ direction is Tess thii 

90 degrees (i.e. 0 to 89), or greater than 270 degrees (i.e., 271 to 360) then 
rt IS a -headwind.- Figure 2 illustrates a -headwind- probler^. Given the wind 
direction, runway heading, wind velocity, and gust (see Figure 2^ the "relative 
wind direction- was determined from subtracting the wind dirertion from thi 

LTn^t^fH^H^ i^'^f^i::^^^^ '^^S^^)- Another rule is that gust valuel 

are not added to headwinds, but are added to crosswinds and taHwinds A 

compo'S^^ '''3"'® ^ ^'"P"^'"^ andToiwind 
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CONSTRAINED 

3pfoiilwnt ptrpfobltniMt 



Gross 




Max. 
XWind 
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320 
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26/23 
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285 
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240 


15 


□ 


205 


15 
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310 


10 
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255 


26/23 
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215 


5 


□ 


290 


3 


□ 


225 


26/23 


□ 




GROSS WEIGHT- 1000 Ifi 

Given grots weight and RCR (rmway condidon reading), 
dttermne the mcodmum aUowabU crofswind for takeoff 
juing the chart above and the toob from the taol menu. 



Figure 1 

Examples of Problem Set for Determining Maximum 
Allowable Crosswind Given Gross Weight and RCR 



The final group of problem sets (7, 8, and 9) required the learner to integrate 
a earlier problem sets. These problems involved determining the maximum 
altowabie crosswind values, computing the wind component values, then deciding 
whether to proceed or abort the takeoff based on the computed information. 

Learning Outcome Measures 

We wanted to assess the acquisition of tutor-specific skills as well as general 
graph reading and interpreting skills that may have been enhanced as a function 
of learning from the tutor. To achieve this end, we created three categories 
of posttests: (a) Basic graph knowledge and skills, (b) Complex graph knowledge 
and SKilis, and (c) Tutor-specific knowledge and skills. Within each catego^r 
several tests measured declarative knowledge and procedural skill acquisition 
separately. All tests were administered on-line at the conclusion of the tutor. 

1. Basic graph knowledge and skills. This test was divided into five parts 
and consisted of 50 items altogether. It was a multiple choice format (with 
• 0 alternative responses to choose from). For example: The horizontal part 
of a graph is called the 7 (correct answer is x-axis). 
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CROSSWIND COMPONENT - KNOTS 



Headvnnd Pmblem 

GIVEN: 



Wind Dir. = 200" 
Runway Heading = 235° 
WindVel. = 15kts. 
Gust = 5kts. 



THUS: 

1. Relative Wind Dir. = 35** 

2. Add Vector at 35" 

3. Intersect wind vel. (15 kts) 
and wind dir. (35°) • — 



4. HWcomp. = 12kts. 



5. CW = Vel. + gust = 20 kts. 

6. Plot vert, line from 20 kts. 
and 35" 



7. Crosswind comp. = 10.5 kts. 



Figure 2 

Seven-Step Solution for Solving Headwind and Crosswind 
Problems from the Wind Components Chart 



2. Complex graph knowledge and skills. We created this test to assess 
performance on more complex graph-related problems. There were three main 
PnSLt 5 /f^T"^'"^ functions Story problems, and Complex graph interpretation, 
nnf fc^lnHini f *° ^'^^ interpretation skills such as 

understanding functional relations and multidimensional graphs. The storv 

Pn°thrflv,ht®'*® ''■^^^^''^ *° ^ analogous to some of the problems encountered 
in the flight engineenng tutor, but without any of the vernacular. There were 

from *°'*'^^* alternatives to choose 

ocofeo l"^^':^^^'^^ ^"^^ ^® created the final posttest to 

assess the depth and tpreadth of knowledge and skills acquired from the tutor 
All three parts comprising this posttest were tutor-specific. The three parts 
measured: Declarative knowledge. Procedural knowledge, and Graph 

TJiF'^T\ ' ""f® ^^'^^ altogether in this posttest. administered in 
a multiple choice format with six alternatives to choose from 
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Error Types 

Ten different types of errors were tallied during the learning process. These 
were classified as either numerical or conceptual errors. Numerical errors 
included: (a) Incorrect HW or TW value entered, (b) incorrect maximum allowable 
crosswind value, (c) incorrect crosswind component computed, (d) omitted data 
in TOLD sheet, and (e) failure to abort an unsafe flight (or choosing to abort 
a safe flight) due to incorrect wind component value (s). Conceptual errors 
were: (a) reversing axes on Wind Components chart, (b) reversing axes on 
Maximum Allowable Crosswind chart, (c) reversing vectors on Wind Components 
chart, (d) interpreting from incorrect curve on the Maximum Allowable Crosswind 
chart, and (e) using an incorrect vector on the Wind Components chart. 



Procedure 

Subjects were tested in groups of approximately 20 persons, and there were 
twenty groups tested, total. Each group spent seven days (about six hours 
per day) in this study. Subjects began the study being tested on basic cognitive 
process measures (not relevant to this paper), and then were randomly assigned 
to one of the learning environments of the tutor. Directly following the completion 
of the tutor, subjects were administered the criterion posttest battery. 



RESULTS 

As expected, subjects in the constrained environment did complete the tutor 
faster than subjects in the extended environment {Constrained M = 8.0 hr, SD 
= 4.0, N = 181; Extended M - 10.6 hr, SD - 4.5, N = 183). An ANOVA on 
these data showed the difference to be significant (F 1,362 - 33.14; p < .001). 

But a more interesting question addressed how well the respective groups 
performed on the outcome measures. Table 1 shows each of the individual 
(and average) posttest scores separated by the two versions of the tutor. 
Even-odd reliabilities of the Posttests were high (r = .91, Posttest 1; r = .91, 
Posttest 2; r = .90, Posttest 3). 

These results were very surprising. In all of the outcome measures, there 
were no differences between the two practice environments. In addition, this 
finding was not a function of differential incoming knowledge. A pretest was 
administered to all subjects before starting the tutor corresponding to the same 
item types as in Posttest 1 (i.e., Basic Graphs). There were no differences 



^Som« indivkMt dropped out before completing eH perle of the study. 
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on any of the individual tests by treatment group, or in the overall pretest 
score {Pretest-Constrained M = 61.8, SD = 20.16, N = 181; Pretest-Extended 
M = 62.6, SD = 18.2, N = 183; F Lsea = 0.19, NS).* 



Table 1. Posttest Scores Separated by Learning Environment 





Test 


Const rainfid 
Iff/ 




B 
1 


OlyniT 


1 


- BASIC GRAPHS 












Graph Knowledge 


60.40 


57.82 


0.81 


NS 




Read Points 


53.45 


52.45 


0.17 


NS 




Read Relations 


69.77 


66.15 


2.01 


NS 




Interpret Points 


62.49 


63.80 


0.34 


NS 




interpret Relations 


52.26 


52.29 


0.00 


NS 




(Average) 


59.67 


58.48 


0.32 


NS 


2 


- COMPLEX GRAPHS 












Functions 


27.82 


30.00 


0.35 


NS 




Story Problems 


56.47 


59.32 


1.99 


NS 




Complex Graphs 


69.44 


70.22 


0.10 


NS 




(Average) 


56.45 


58.37 


0.84 


NS 


3 


- TUTOR SPECIFIC 












Declar. Knowledge 


61.02 


60.89 


0.00 


NS 




Proced. Knowledge 


48.08 


48.21 


0.00 


NS 




TOLD sheet 


62.71 


65.86 


2.21 


NS 




(Average) 


59.12 


60.83 


0.65 


NS 



The next set of questions concerned differences between groups as a 
ftjnction of latencies and errors per problem set. Figure 3 shows the data for 
the three groups of problems. Problem sets 1, 2, and 3 all involve figuring 
out maximum allowable crosswind; sets 4, 5, and 6 involve computing headwind, 
tailwind, and crosswind values; and finally, sets 7, 8, and 9 involve integrating 
the prior problem sets. The first row of graphs represents solution time by 
problem. The second row represents error data (total errors per problem). 



'In KMition, Mch potttMt wu r«grMMd againtt the mMn pnrtMt score and the rmkiuait MV«d. Comparing outcomM wHh 
ttMM fMiduaKzod poMMt wotm again ahowwl no aignificwtt «t«w«no«a batwaan oonditiona. 



Pn)MfflS«lt1,2.3(iiax.CroMWtnd) ProWMnS*»4,5,«(HW,TW,XW) ProbtomS*«7,l,e{CoiiiptMlnoTOLD) 




Probtam8tt»1,2,3(ll«.CroMwind) Probkm Srtt 4, 5, 6 (HW, TW, XW) Problem Seti 7, 8, 9 (Computing TOLD) 



MriEmrt 



TMEnn 




t 2 3 4 S I 7 t t 10 11 12 



1 2 3 4 S I 7 I t 10 11 12 



1 2 1 4 $ I 7 I 0 10 11 12 



Figure 3 

Latency and Error Data Across Problem Sets 



For the easiest problem sets, there were no strong differences between the 
two groups for either latency or error data. But the second group of problem 
sets shows some separation between learning curves whereby subjects in the 
constrained version began to take longer per problem compared to those in 
the extended version. Finally, for the complex problems (sets 7, 8, and 9), 
the gap has widened considerably for the time data. Subjects appeared to have 
adjusted their learning times upward in the constrained version. The mean 
time to learn these problem sets for subjects in the constrained version was 
54.24 minutes (SD = 22.13, N = 177), and the mean time for subjects in the 
extended version was 35.77 minutes (SD = 15.34, N = 178). This finding 
represented a significant difference between the learning environments (F 1353 
= 83.61, p < .001). 

14 



ERIC 



8 



In terms of the error data, only "total errors" were plotted. Again, the same 
pattern of differences between learning environments emerged. The number 
of errors made during the easiest problems (sets 1, 2, and 3) was about the 
same for the two learning environrrsents. For the next grouping of problems 
(sets 4, 5, and 6), a slight separation between groups began to appear, but 
this difference was not significant. By the final group of problems (sets 7, 8, 
and 9), subjects in the constrained environment were making more errors than 
subjects in the extended environment. This difference was statistically significant 
(F 1.353 = 25.68, p < .001). 

To determine any differences in the types of errors committed, the error 
data were separated into either "numerical" or "conceptual" errors and we 
computed a proportion: numerical divided by conceptual errors. To test for 
differences between the two practice environments, we computed a one-way 
Analysis of Variance (ANOVA) on this proportion by environment. Results 
showed that subjects in the constrained environment had a higher proportion 
of numerical to conceptual errors (M = 2.69; SD = 1.55; N = 181) compared 
to subjects in the extended environment (M = 2.12; SD = 0.88; N = 183). 
This difference was significant (F 1,362 = 18.82; p < .001). 



DISCUSSION 

Practice effects were investigated in relation to overall learning time, learning 
outcome, and parameters of skill acquisition (latencies and errors). The first 
hypothesis was supported: subjects in the constrained environment did require 
significantly less time to complete the tutor compared to subjects in the extended 
environment. The second hypothesis, that subjects in the extended environment 
would perform better on outcome measures than subjects in the constrained 
environment, was not supported and the surprise finding was that there were 
no differences on any of the outcome measures between the two groups. This 
finding may, in part, be explained by the third finding that subjects in the 
constrained environment gradually increased learning times per problem, 
presumably to compensate for having so few practice problems to solve (thus, 
hypothesis 3 was supported). Finally, it was hypothesized that subjects in the 
constrained environment would make more conceptual than numerical errors. 
The opposite was found. But this finding can actually account for the other 
unexpected finding about the comparability of outcome measures between groups. 
Because subjects in the constrained environment were investing more time per 
problem, especially within the most difficult problem sets, they were able to 
extract the conceptual issues from the problems, but it cost them additional 
time per problem. Thus, the errors they made were less conceptual and more 
numerical (i.e., a higher proportion of numerical/conceptual errors compared to 
the subjects in the extended environment). If they had made more conceptual 
errors (in relation to numerical errors), then we would expect to see differences 
in outcome performance. 
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In conclusion, less practice (1/4 the problems) can actually result in comparable 
learning outcomes to more extensive practice environments. The cost is in 
terms of more time invested per problem as well as less precise computa^'ons 
(i.e., numerical errors). But numerical errors can have severe consequences 
in the real world. Thus, these data suggest that the tutor should remediate 
numerical errors, which it currently does not. Additional research is indicated 
to determine the boundary conditions under which we can see these same 
results. In other words, just how low can we go? Furthermore, research is 
planned to test if differences in retention could be detected between the two 
groups after a significant period of time has elapsed (e.g., 9 months to 1 year). 
Findings may show that benefits from the extended condition show up in terms 
of better retention compared to subjects learning from the constrained environment. 
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